As the number of online connected users, devices and applications grow it is becoming increasingly difficult for new applications to attract, retain, and ultimately monetize new users. The current processes for optimizing applications may produce sub-optimal results, may be time consuming, and may be inefficient in terms of both time and resources. The current practice is to record and monitor key performance indicators (KPIs) in an effort to identify KPIs to attempt to improve. Common KPIs include (but are not limited to) daily active users, total paying users, conversion rates, average revenue per daily active user, etc. When a KPI to improve is agreed upon, then a hypothesis on why the KPI is underperforming and recommendations for possible solutions may be formed. In many cases, teams of data analysts analyze log files generated by the application to aid in discovery of possible solutions. Eventually, a solution is proposed and presented to the software development team. If the team accepts this proposal, they will modify the application. In many cases these changes will be included in the next release of the software. In some cases, the development team may choose to roll out the changes to a small group of users so they can compare the KPI impact of the changes to a control group that does not have the changes. This process is repeated indefinitely until the KPIs reach their business targets or the continued support and development of the application no longer meets the goals of the project. Refer to FIG. 1 for an example of this process applied to a video game application running in a social network.